Parsing, Word Associations and Typical Predicate-Argument Relations

There are a number of coUocational constraints in natural languages that ought to play a more important role in natural language parsers. Thus, for example, it is hard for most parsers to take advantage of the fact that wine is typically drunk, produced, and sold, but (probably) not pruned. So too, it is hard for a parser to know which verbs go with which prepositions (e.g., set up) and which nouns fit together to form compound noun phrases (e.g., computer programmer). This paper will attempt to show that many of these types of concerns can be addressed with syntactic methods (symbol pushing), and need not require explicit semantic interpretation. We have found that it is possible to identify many of these interesting co-occurrence relations by computing simple summary statistics over millions of words of text. This paper will summarize a number of experiments carried out by various subsets of the authors over the last few years. The term collocation will be used quite broadly to include constraints on SVO (subject verb object) triples, phrasal verbs, compound noun phrases, and psycholinguistic notions of word association (e.g., doctor~nurse). 1. Mutual Information Church and Hanks (1989) discussed the use of the mutual information statistic in order to identify a variety of interesting linguistic phenomena, ranging from semantic relations of the doctor/nurse type (content word/content word) to lexico-syntacfic co-occurrence constraints between verbs and prepositions (content word/function word). Mutual information, l(x;y), compares the probability of observing word x and word y together (the joint probability) with the probabilities of observing x and y independently (chance). l(x;y) -= log 2 P(x,y) e(x) e(y) If there is a genuine association between x and y, then the joint probability P(x,y) will he much larger than chance P(x) P(y), and consequently l(x;y) >> 0, as illustrated in the table below. If there is no interesting relationship between x and y, then P(x,y) = P(x) P(y), and thus, I(x;y) = 0. If x and y are in complementary distribution, then P(x,y) will be much less than P(x) P(y), forcing l(x;y) << 0. Word probabilities, P(x) and P(y), are estimated by counting the number of observations of x and y in a corpus, f (x) and f(y), and normalizing by N, the size of the corpus. Joint probabilities, P(x,y), are estimated by counting the number of times that x is followed by y in a window of w words, fw(x,y), and normalizing by N (w 1). 2 1. The paper was previously presented at International Workshop on Parsing Technologies, CMU, 1989. 2. The window size parameter allows us to look at different scales. Smaller window sizes will identify fixed expressions (idioms), noun phrases, and other relations that hold over short ranges; larger window sizes will highlight semantic concepts and other relationships that hold over larger scales.